Telemedicine Service Experience Questionnaire for Chinese Outpatients: Development and Validation Study

Introduction

With the rapid development of information technology and the popularization of the internet, telemedicine services are becoming an innovative approach in the medical field [1] and are being used in a variety of settings, including psychiatric counseling [2], home health care services [3], intensive care units [4], and emergency stroke care [5], to provide patients with efficient and convenient medical care. In recent years, the worldwide COVID-19 outbreak has accelerated the spread and adoption of telemedicine services in health care [5]. Faced with the constraints of outbreaks and the need for prevention and control, telemedicine services protect patients, clinicians, and communities from exposure. In addition, they provide a platform where physicians and patients can interact using smartphones or webcam-enabled computers, regardless of time or location [6]. Particularly in China, a vast country with rural areas and dispersed populations, telemedicine services hold great potential and advantages in addressing the unequal distribution of health care resources and improving the quality of medical services [7]. The Chinese government has explicitly included telemedicine services as a key area in the development of the national health care system. In 2018, the National Health Commission of China launched the Action Plan for Further Improvement of Medical Services (2018‐2020). The program encourages medical institutions and research institutes to strengthen innovative research and development and vigorously promotes the integration of telemedicine technology with medical resources to facilitate the gradual development of telemedicine [8].

As an emerging health care model, patient satisfaction with telemedicine services is an important indicator of the quality of telemedicine services and an important driving force for the development of telemedicine services. Collecting and analyzing patient feedback can help medical institutions and decision-makers to comprehensively understand patients’ demands and expectations, improve and optimize service quality, attract more patients to accept telemedicine services, and promote their wide application in health care. Studies have shown that the use of telemedicine services will only expand if patients and health care professionals are at least as satisfied with telemedicine services as they are with in-person visits [9]. In addition, highly satisfied patients are likely to have higher medication adherence, leading to better health outcomes [10].

Several studies have been conducted to explore the satisfaction with telemedicine services, such as the Telemedicine Satisfaction and Usefulness Questionnaire [11], Telemedicine Satisfaction Survey [12], and Telehealth Usability Questionnaire [13]. Nevertheless, telemedicine-specific, psychometrically validated patient-reported experience instruments tailored to Chinese outpatient workflows remain limited. In China, the scales currently available for measuring telemedicine service satisfaction are mainly based on the extension and improvement of traditional health care service satisfaction scales and lack sufficient consideration of the specific characteristics and unique features of telemedicine services [14]. For example, these scales may not fully capture telemedicine-specific processes (eg, digital information guidance, postconsultation payment and medication delivery, and privacy or security concerns).

To address this gap, we developed the Telemedicine Service Experience Questionnaire (TSEQ), using various items adapted for patient satisfaction surveys among Chinese outpatients. Our primary objective in this paper was to assess the appropriateness of the TSEQ scale as an instrument to measure patient satisfaction across a broad range of outpatient populations in China. Rather than measuring technology usability alone, the TSEQ is designed to capture the service-process experience across the entire outpatient telemedicine journey, including preconsultation guidance, interaction quality, and postconsultation processes (eg, payment and medication delivery), which are particularly salient in platform-based telemedicine services. Accordingly, the objectives of this study were to (1) develop the TSEQ for Chinese outpatient telemedicine services; (2) evaluate its underlying factor structure using exploratory and confirmatory factor analyses; and (3) assess its reliability and validity, including internal consistency and convergent validity.

Methods

Data Collection and Sample

Our research focuses on users of hospital apps at Peking Union Medical College Hospital, a leading medical institution renowned for its comprehensive health care services and advanced medical technology. Data were collected through a web-based questionnaire from July 1, 2021, to August 31, 2021. Eligible participants were adults (aged ≥18 years) who completed a telemedicine consultation and provided informed consent. We excluded responses with substantial missing data, implausibly short completion times, or duplicate submissions where applicable. Convenience sampling was used, which is common in psychometric validation studies. In this study, patients who fully completed the TSEQ questionnaire were included (N=3337). The survey window was selected because telemedicine services at the study hospital had reached stable operational routines and sufficient user volume for psychometric evaluation. As this study focuses on instrument development and validation, the 2021 dataset provides an appropriate foundation to examine the factor structure and reliability and validity of the TSEQ.

Ethical Considerations

A web-based informed consent was obtained before the survey via WeChat. Participants were allowed to withdraw from the survey at any time. All interviews of the survey were kept confidential and anonymous. This study was approved by the Peking Union Medical College Hospital Ethics Committee (S-k1545). Responses were collected anonymously (or deidentified), and no directly identifiable personal information was stored. Data were stored on secure servers with restricted access limited to the research team. Participants did not receive any compensation for participation in this study.

Measures and Scale Development

The TSEQ was adapted from the Chinese Patient Experience Questionnaire (CPEQ; refer to Multimedia Appendix 1 for details) [15]. The CPEQ consists of 2 sections: part A includes 9 questions that collect basic demographic information of the participants, such as age, sex, employment status, and type of insurance and part B comprises 22 questions across 5 dimensions and is designed to comprehensively evaluate patient experience. On the basis of expert discussions on the CPEQ, we developed a 15-item questionnaire for telemedicine service experience, focusing on self-efficacy, information guidance, and humanistic care. Every question was assessed using a 5-point Likert scale (5=strongly agree, 4=agree, 3=unsure, 2=disagree, and 1=strongly disagree). The newly formed TSEQ is presented in Multimedia Appendix 2.

The validation and reliability of the TSEQ are the cornerstones of our research’s second objective. We assessed the instrument’s internal consistency and test-retest reliability, alongside its validity, to ensure it measures the intended constructs. An open-ended question was included at the end of the questionnaire to capture valuable, unstructured feedback from the respondents.

Statistical Analysis

Descriptive statistics were used to summarize and describe the demographic characteristics of the participants, such as age, sex, occupation, and education. Categorical variables (eg, sex, occupation, and education) were presented as numbers (percentages), and continuous variables (eg, age) were reported as mean (SD).

Exploratory Factor Analysis

Exploratory factor analysis (EFA) was first performed to explore the number of factors and the factor loading of each item using maximum likelihood estimation, which is the default estimator in Mplus (version 7.0; Muthén & Muthén). Before the factor analysis, Bartlett test of sphericity was usually used to examine whether these variables were independent of each other and suitable for factor analysis. In this study, we computed 1-factor, 2-factor, 3-factor, and 4-factor measurement models using EFA. To identify the model structure, we used a series of indicators such as χ², df, Tucker-Lewis index (TLI), comparative fit index (CFI), Akaike information criterion, Bayesian information criterion [1,16], standardized root mean square residual (SRMR), and root mean square error of approximation (RMSEA) to evaluate the goodness-of-fit statistics. Acceptable model fit was defined by the following criteria: TLI>0.90, CFI>0.90, SRMR<0.05, and RMSEA<0.08 [17-19].

During EFA, items were considered for removal if they exhibited low factor loadings (eg, <0.40), substantial cross-loadings, low commonalities, weak item-total correlations, or conceptual redundancy. The refined structure was then tested using confirmatory factor analysis (CFA), and model fit indices guided the final scale specification.

Model Validation Using CFA

Following EFA, we conducted the CFA to verify whether the hypothesized structure from EFA was consistent and valid. CFA is a statistical technique used to evaluate the effectiveness of the measurement model and examine the relationship between measured variables and underlying latent constructs or factors. In this study, first-order factors were verified for the dimensions of telemedicine service experience, and a second-order model was established to explain the relationships among multiple first-order factors. We computed and compared the overall fit coefficients, χ²/df, RMSEA, CFI, TLI, and SRMR of CFA models to remove inappropriate items and confirm the ideal model results. Moreover, the model fit was evaluated using the standardized factor loadings and R².

Reliability and Validity

To assess the reliability of CFA-confirmed models, we further computed Cronbach α and split-half reliability coefficients (SRCs) for the total questionnaire, final questionnaire, service efficiency (F1), post-treatment (F2), information guidance (F3), and humanistic care (F4). With regard to the reliability of a research instrument, statistical theory has determined that Cronbach α should be >0.7 [20,21]. Correlation analysis was used to investigate the correlation between the first-order factors and Chinese patients’ overall satisfaction and willingness to recommend. Finally, a validity test was conducted to verify the validity of the newly proposed Chinese adaptation of the patient experience measurement instrument by estimating the factor loadings to examine the relationships between observed variables and latent constructs that we wanted to measure. Standard factor loadings >0.70 were considered the criterion for convergent validity [22].

Results

Sociodemographic Characteristics

Among the 3337 participants, the mean age was 45.3 (SD 17.8) years. The majority were female (n=2182, 65.4%), had completed a bachelor’s degree or higher education level (n=1827, 54.8%), paid using health insurance (n=2292, 68.7%), and were seeking medical care outside their local area (n=1372, 41.1%; Table 1).

Table 2 presents the performance scores for the 15 individual items of the TSEQ in this study. These items primarily focus on various aspects of hospital services, including waiting times, service convenience, the communication style of physicians, and the respect and privacy protection afforded to patients during their medical visits. The average scores for each item generally range between 4.2 and 4.6, indicating that most respondents are satisfied with the services provided by the hospital and that the scores demonstrate relatively high consistency. These findings support further development of the scale for the EFA and the CFA.

EFA Findings

The results of the EFA indicated that the Bartlett test of sphericity yielded a χ² value of 785.966 (df=51; P<.001; Kaiser-Meyer-Olkin value=0.950; Table 3), suggesting that the dataset used was suitable for factor analysis. Through this analysis, we successfully established a 4-factor structure, with these factors collectively explaining 78.49% of the total variance (Multimedia Appendix 3). This indicates that the TSEQ tool exhibits good structural validity, accurately reflecting various aspects of patient experiences in remote health care services.

Of the 15 items, 14 loaded meaningfully onto 1 of the 4 factors, with factor loadings ranging from 0.38 to 0.89. However, one item (item 12: “Physicians discussed my condition and care patiently”) had cross-loadings on factors 2 and 4. Despite this, considering the importance of this item, we did not remove it in the EFA stage. The EFA resulted in a 4-factor solution that explained 79.00% of the variance (Table 4).

CFA Findings

In the CFA, it was observed that the modification index for item 12 was excessively high across the 4 factors (>200; Multimedia Appendix 4). After its removal, the model fit improved in the CFA model. Therefore, the final model excluded item 12 (Table 5).

Given the 4 factors derived from EFA, an interpretation focusing on items with higher factor loadings for each factor, and their comparison to the TSEQ, was performed by the panel (Table 6). Factor 1 was labeled “Service Efficiency,” as it consisted of 6 items dealing with the convenience of telemedicine services. The second factor, consisting of 2 items, was called “Post-treatment,” as the content involves the time spent on medication dispensing and payment processes. Factor 3 comprised 3 items that captured information-related elements of telemedicine services; therefore, the title of this factor was “Information Guidance.” Finally, the 3 items in factor 4, which involve the physician’s’ attitudes toward patients, were labeled “Humanistic Care.”

Internal Consistency and Validity

As presented in Table 7, the final instrument obtained a Cronbach α coefficient of 0.946 and an SRC value of 0.872 for the total sample. The 4 factors—service efficiency, post-treatment, information guidance, and humanistic care—each obtained Cronbach α coefficients ranging from 0.840 to 0.962, and their SRC values were 0.877 (6 items), 0.840(2 items), 0.965 (3 items), and 0.892 (3 items), respectively. These results demonstrate that the TSEQ has good internal consistency.

The correlation coefficients between overall satisfaction and the 4 factors—service efficiency, post-treatment, information guidance, and humanistic care—were 0.941, 0.781, 0.857, and 0.901, respectively. All the correlations were statistically significant at the .01 level and demonstrate good internal validity. Moreover, as the average variances extracted values were >0.5 and the composite reliability values were >0.7, the TSEQ scale demonstrated high convergent validity (Table 7).

TSEQ Scoring

The TSEQ measures each item on a 5-point Likert-type scale, ranging from 1 (“strongly disagree”) to 5 (“strongly agree”). Calculating a total score involves summing the response scores for all 14 items of the TSEQ. The total score ranges from 14 to 70, with higher scores indicating higher levels of telemedicine service experience. The Chinese version of the questionnaire is shown in Multimedia Appendix 5.

Discussion

Principal Findings

This study developed and validated the TSEQ, an instrument for measuring telemedicine service experience in Chinese outpatients, based on the CPEQ [15]. In this study, EFA produced a revised 4-factor solution for the TSEQ, leading to modifications in the 5 domains of the CPEQ. Telemedicine services offer convenient medical solutions and effectively tackle the issue of uneven distribution of medical resources, especially in remote and rural areas [23,24]. The global COVID-19 pandemic has recently accelerated the adoption and expansion of telemedicine [25,26]. Importantly, our study does not aim to re-establish the well-known advantage that telemedicine reduces geographic barriers. Instead, it contributes a validated, telemedicine-specific patient experience instrument and highlights which service-process dimensions matter in Chinese outpatient telemedicine workflows. In particular, the emergence of a distinct “post-treatment” domain (eg, payment timing and medication delivery) reflects operational features that are often underrepresented in traditional in-person satisfaction tools and provides actionable targets for quality improvement. Our findings contribute to global efforts to standardize patient-reported evaluation of telemedicine quality. Although the TSEQ was developed for Chinese outpatient workflows, its domain-based structure may inform telemedicine service monitoring in other settings, subject to appropriate cross-cultural adaptation and validation.

In this analysis, the final instrument comprises 4 factors with 14 items, designed to evaluate patient satisfaction with telemedicine services and identify areas for service improvement. The EFA supported the viability of a 4-factor model. Compared to previous studies, the EFA uniquely identified “payment timing (item 5)” and “medicine delivery (item 6)” as a new single factor, while grouping “It was convenient to make an appointment for a planned examination (item 7)” and “Medication instruction services provided by the hospital can meet needs (item 8)” under the first factor, service efficiency. Notably, the item “I was treated with respect and dignity during this visit (item 12)” was originally categorized under “Humanistic Care” in the CPEQ framework but was removed in this study. This change may be attributed to the fact that patients using telemedicine services are more focused on the quality and efficiency of the service and may not experience significant nonverbal communication cues such as eye contact, body language, and tone of voice, which are typically present during face-to-face interactions with health care professionals [24,27].

Additionally, although “privacy and security risks (item 13)” showed lower factor loadings in the EFA, these aspects were still included in the TSEQ due to their critical importance in the telemedicine context [28,29], where patient privacy and security are paramount. Telemedicine also introduces challenges, including technology barriers, digital literacy gaps, privacy or security concerns, and constrained clinician-patient communication. By quantifying patient experience across distinct service domains, the TSEQ can help hospital identify where telemedicine workflows fail to meet expectations and prioritize targeted improvements.

Furthermore, CFA validated the 4-factor structure of the questionnaire. According to the pre-established CFI, and RMSEA thresholds, the second-order model can be considered acceptable. Only the χ² test revealed a bad fit for the second-order model, likely due to the large sample size [30]. All standardized factor loadings were statistically significant (all P<.001) and ranged from 0.729 to 0.953. On the basis of the CPEQ [15], our study has introduced a second factor, defined as “post-treatment,” further demonstrating that the efficiency of posttelemedicine services also affects overall patient satisfaction. Analysis of the internal consistency of the Chinese version of the TSEQ showed that the Cronbach α of the entire scale was 0.95, indicating excellent scale reliability. All scales demonstrated good internal consistency, with Cronbach α of >0.80, indicating moderate to good test-retest reliability [30,31]. Moreover, CFA revealed strong correlations among the 4 factors, and further exploration of the relationships among the 4 factors confirmed satisfactory convergent validity based on the average variances extracted and composite reliability values [32,33].

The majority of participants in our study were female (2182/3337, 65.4%) and had completed bachelor’s degree or higher education level (n=1827, 54.8%). Nearly half (n=1372, 41.1%) of the participants were seeking medical care outside their local area. The composition of our study population underscores the broad relevance of our findings. This diversity is essential for understanding the multifaceted patient experiences within telehealth services, emphasizing the need for inclusive and accessible health care solutions that cater to varied demographic segments [30,33].

The TSEQ offers actionable value for routine quality monitoring and continuous improvement in outpatient telemedicine services. In practice, health care professionals and platform administrators can integrate the TSEQ as a brief postvisit feedback tool embedded within the telemedicine workflow (eg, immediately after consultation or after completion of postconsultation steps). Domain-level scores can be visualized in a dashboard to identify specific bottlenecks and track trends over time, enabling a closed improvement loop of “measure-diagnose-intervene-reassess.” For example, low scores in interaction-related domains may prompt targeted clinician training (eg, communication clarity and shared decision-making in remote settings), whereas low scores in the post-treatment domain can inform workflow redesign for payment, prescription processing, and medication delivery or collection. The TSEQ can also support benchmarking across departments, specialties, or service lines to prioritize resource allocation and evaluate the impact of operational changes. At the policy level, aggregated TSEQ indicators may complement use metrics by capturing patient-centered service quality; helping health systems move from “telemedicine adoption” to “telemedicine performance”; and guiding standards for service transparency, privacy protection, and equitable access for digitally disadvantaged groups.

Limitations

Several limitations should be acknowledged. First, this study relied on self-reported survey responses collected immediately after telemedicine use, which may be subject to social desirability bias, recall bias, and common method variance. Second, participants were recruited using a convenience sampling approach from a single hospital setting, which may limit generalizability to other regions, health care tiers, and telemedicine platforms, especially where service workflows, payment pathways, and medication dispensing or delivery arrangements differ. Third, the sample characteristics may not fully represent telemedicine users with limited digital access or lower digital literacy; therefore, our estimates of service experience may be upwardly biased for populations facing stronger technology barriers. Fourth, data were collected during mid-2021, when telemedicine use and patient expectations were shaped by the COVID-19 pandemic context; experience patterns may shift in postpandemic periods when patients have broader choices between in-person and remote visits. Fifth, the cross-sectional design precludes assessment of temporal stability (eg, test-retest reliability), responsiveness to service improvements, and longitudinal predictive validity. Finally, although we established overall psychometric properties, formal measurement invariance and differential item functioning across key subgroups (eg, age, sex, education, allopatry treatment, and previous telemedicine experience) were not the primary focus of this manuscript and warrant further validation in multisite and postpandemic samples.

Conclusions

In conclusion, this study developed and psychometrically validated the TSEQ for Chinese outpatients. The final instrument demonstrated a stable multidomain structure, indicating that it can be used to capture key dimensions of patients’ telemedicine service experience in routine outpatient care. By providing domain-specific scores, the TSEQ enables health care institutions and platform administrators to identify actionable targets for quality improvement, particularly in end-to-end workflow components that are salient in telemedicine settings (eg, communication processes and postconsultation services). Future research should validate the TSEQ in multisite and postpandemic samples, assess test-retest reliability and responsiveness to service changes, and examine measurement invariance or differential item functioning across key subgroups to support broader implementation and benchmarking.